feat: Composio sync harness + connect flow + durable memory viewer#71
Conversation
Adds a runnable example that validates a Composio API key end-to-end: Phase 1 lists connected accounts (v3 /connected_accounts) to confirm the key and discover toolkits + connection ids; Phase 2 runs each TinyCortex Composio pipeline (gmail/github/linear/notion/clickup/slack) twice against real Composio using in-memory sinks, grading ingest, actions, cost, cursor advance, taint tagging, and idempotency, then prints a PASS/FAIL summary and exits non-zero on failure. Run: COMPOSIO_API_KEY=... cargo run --example composio_harness --features sync Claude-Session: https://claude.ai/code/session_01XLggQ7yzXY39k7nQfy2jvP
The harness now auto-loads a gitignored .env from the working dir (real process env still overrides it), and ships a committed .env.example template documenting every supported variable. Claude-Session: https://claude.ai/code/session_01XLggQ7yzXY39k7nQfy2jvP
Sync pipelines hand SkillDocuments to a host-owned SkillDocSink; the crate previously shipped only in-memory sinks, so nothing a sync run ingested survived the process. KvSkillDocSink persists each document as one kv_namespace row (namespace skilldoc:<toolkit>, key document_id), letting a host or debug viewer browse ingested memory without re-running sync. KV write sanitization means the persisted content is already scrubbed of secrets/PII. Claude-Session: https://claude.ai/code/session_01AtynpEKn3QZJKZYvL8uDNR
When TINYCORTEX_WORKSPACE is set, the harness writes ingested SkillDocuments through the durable KvSkillDocSink and drops a sync_manifest.json (per-toolkit results + the full event stream) so the memory viewer can inspect what a sync run produced. Confined to the Captures/Phase-2 region to stay clear of the in-flight connect-flow work in Phase 1. Claude-Session: https://claude.ai/code/session_01AtynpEKn3QZJKZYvL8uDNR
Add src/memory/sync/composio/connect.rs owning the reusable, host-agnostic
pieces of the Composio login/connect flow:
- EntityStore: per-toolkit entity-id (user_id) persistence to a JSON file so
re-runs reuse the same Composio entity instead of orphaning connections.
- Status classification (active/terminal) and response-field extraction for
the v3 connected-account lifecycle.
- Thin async wrappers over three verified v3 endpoints:
GET /auth_configs?toolkit_slug=, POST /connected_accounts/link,
GET /connected_accounts/{id}.
Endpoints/fields verified against the Composio SDK source and v3 API reference.
No secrets logged; error paths discard raw response bodies. Covered by
connect_tests.rs (10 tests).
Claude-Session: https://claude.ai/code/session_01AtynpEKn3QZJKZYvL8uDNR
Add Phase 1.5 to the composio_harness example: for toolkits requested via
COMPOSIO_TOOLKITS that have no ACTIVE connected account (none discovered, or a
terminal FAILED/EXPIRED/REVOKED status), initiate a Composio Connect login:
- resolve the auth-config id (COMPOSIO_<TOOLKIT>_AUTH_CONFIG_ID override, else
GET /auth_configs?toolkit_slug=), scoped to a per-toolkit entity id remembered
in the gitignored .composio-harness.json;
- create a connect link, print the OAuth URL, and poll
GET /connected_accounts/{id} until ACTIVE, terminal, or COMPOSIO_CONNECT_TIMEOUT_SECS;
- on success feed the new connection into the existing Phase 2 sync loop.
Skipped when COMPOSIO_TOOLKITS is unset so the default run stays non-interactive.
Phase 2 sync loop, Captures sinks, and print_report are untouched. Documents the
new env vars in the harness doc-comment and .env.example; gitignores the store.
Exercise: COMPOSIO_API_KEY=... COMPOSIO_TOOLKITS=gmail \
cargo run --example composio_harness --features sync
Claude-Session: https://claude.ai/code/session_01AtynpEKn3QZJKZYvL8uDNR
A local, read-only debug UI (viewer/) whose server-side code opens the workspace's SQLite stores and reads sync artifacts: overview, skill-doc browser with filter/search + detail, memory-tree chunks, entities, and the last sync run manifest. Uses better-sqlite3 read-only; no network access. examples/seed_memory.rs seeds a workspace with sample synced memory so the viewer is demoable without a live Composio key. Cargo excludes viewer/ from the published crate. Claude-Session: https://claude.ai/code/session_01AtynpEKn3QZJKZYvL8uDNR
seed_memory now builds an actual summary tree (chunks -> L0 -> L1/L2) via TreeFactory + the offline ConcatSummariser across two source scopes, so the viewer's memory-graph and memory-tree views have real data to render without a live sync. Skill-doc + manifest seeding is unchanged. Claude-Session: https://claude.ai/code/session_01AtynpEKn3QZJKZYvL8uDNR
Adds a /graph page: server-side memoryGraph() reads mem_tree_trees/summaries/ chunks and derives a node/edge graph (root -> source -> summary levels -> chunks), rendered client-side with d3-force + SVG. Matches OpenHuman's memory graph encoding (color/radius by node kind and summary level), with pan, zoom, node drag, hover detail, auto-fit, and a Fit button. Every top-level summary is attached to its source so multi-root trees stay one connected component. Claude-Session: https://claude.ai/code/session_01AtynpEKn3QZJKZYvL8uDNR
# Conflicts: # examples/composio_harness.rs
Composio v3 rejects tool execution with connected_account_id unless the account's user_id is also sent (error 1811, ActionExecute_ConnectedAccountEntity IdRequired). The harness now captures each account's user_id during discovery (and from the connect flow) and scopes that toolkit's transport to it, falling back to COMPOSIO_ENTITY_ID. Verified live: github sync went from HTTP 400 to PASS with 50 documents ingested. Claude-Session: https://claude.ai/code/session_01AtynpEKn3QZJKZYvL8uDNR
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 21 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (2)
📒 Files selected for processing (30)
Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8c53433e1c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| { | ||
| continue; | ||
| } | ||
| if connection.status.as_deref().is_some_and(status_is_terminal) { |
There was a problem hiding this comment.
When COMPOSIO_TOOLKITS names a toolkit whose discovered account is still INITIATED, INITIALIZING, or INACTIVE, this check only drops terminal states, so the toolkit is added to selected and Phase 1.5 is skipped; Phase 2 then tries to execute tools on a non-ACTIVE account and fails instead of initiating/polling the connect flow. Treat non-active discovered statuses as absent, while still allowing explicit env overrides, so requested-but-not-active accounts go through login.
Useful? React with 👍 / 👎.
| @@ -0,0 +1,799 @@ | |||
| //! Composio connection + memory-sync harness. | |||
There was a problem hiding this comment.
This new Rust source is 799 lines, which violates the repository guideline in AGENTS.md: “Avoid letting any source file grow beyond 500 lines of code.” That makes future changes to the harness harder to review and maintain; please split the sinks, connection helpers, reporting, and persistence into focused modules before landing.
Useful? React with 👍 / 👎.
1 participant
Summary
Adds an end-to-end Composio connection + memory-sync harness, makes sync
output durable, and ships a local memory viewer web app to observe/debug
it — including a force-directed view of the summary tree. Also fixes a Composio
v3 breakage that blocked all live sync.
Validated live against a real Composio account: GitHub sync goes from HTTP 400
→ PASS, 50 documents ingested and rendered in the viewer.
What's included
Sync engine (library)
KvSkillDocSink— the first durableSkillDocSink. Sync previouslyhanded documents to a host-owned sink and the crate shipped only in-memory
ones, so nothing survived a run. Persists each
SkillDocumentas onekv_namespacerow (KV write-sanitization scrubs secrets/PII).composio/connect.rs) — per-integrationentity-id (
user_id) store + OAuth login flow (/auth_configs,/connected_accounts/link, status polling). Host-agnostic, unit-tested.user_id— v3 rejects tool execution unless the account'suser_idis sent withconnected_account_id(error 1811). The harness nowcaptures each account's
user_idat discovery and scopes the transport to it.Harness (
examples/composio_harness.rs)toolkits, Phase 2 twice-tick memory sync with grading (records, actions, cost,
taint, cursor, idempotency).
.envsupport, andTINYCORTEX_WORKSPACEto persist ingested docs +sync_manifest.jsonfor the viewer.examples/seed_memory.rs— builds a real summary tree offline (no key) sothe viewer/graph are demoable.
Memory viewer (
viewer/)(force-directed root → source → summary → chunk, matching OpenHuman's encoding;
pan/zoom/drag/fit), Memory tree, Entities, Sync runs.
Docs
TESTING.md— how to run the tests, harness, seed, and viewer end-to-end.Testing
cargo test --features sync— 1158 + integration tests pass;cargo fmt/clippyclean.viewer:npx tsc --noEmitclean.COMPOSIO_TOOLKITS=github cargo run --example composio_harness --features sync→
HARNESS PASS, 50 docs; verified in the viewer.Notes
separate ingest pass, so
/graphand/treeare populated by the seed example(or a real ingest) while
/docsand/runsreflect live sync.https://claude.ai/code/session_01AtynpEKn3QZJKZYvL8uDNR